{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Sentiment Analysis with TensorFlow 2\n",
    "\n",
    "Amazon SageMaker provides both built-in algorithms and an easy path to train your own custom models.  Although the built-in algorithms cover many domains (computer vision, natural language processing etc.) and are easy to use (just provide your data), sometimes training a custom model is the preferred approach.  This notebook will focus on training a custom model using TensorFlow 2.  \n",
    "\n",
    "Sentiment analysis is a very common text analytics task that involves determining whether a text sample is positive or negative about its subject.  There are several different algorithms for performing this task, including statistical algorithms and deep learning algorithms.  With respect to deep learning, a 1D Convolutional Neural Net (CNN) is sometimes used for this purpose.  In this notebook we'll use a CNN built with TensorFlow 2 to perform sentiment analysis in Amazon SageMaker on the IMDB dataset, which consists of movie reviews labeled as having positive or negative sentiment. Three aspects of Amazon SageMaker will be demonstrated:\n",
    "\n",
    "- How to use a SageMaker prebuilt TensorFlow 2 container with a custom model training script similar to one you would use outside SageMaker. This feature is known as Script Mode.  \n",
    "- Local Mode training:  this allows you to prototype and test your code on a low-cost notebook instance before creating a full scale training job on more powerful and expensive instances.\n",
    "- Hosted training: for full scale training on a complete dataset.  \n",
    "- Distributed training:  using a single, multi-GPU instance to speed up training.  \n",
    "- Batch Transform for offline, asynchronous predictions on large batches of data. \n",
    "\n",
    "#  Prepare the dataset\n",
    "\n",
    "We'll begin by loading the reviews dataset, and padding the reviews so all reviews have the same length.  Each review is represented as an array of numbers, where each number represents an indexed word.  We'll also pad shorter reviews to match a maximum specified length."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install --user -qU 'sagemaker[local]'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: matplotlib in /Users/cfregly/miniconda3/envs/localsm/lib/python3.7/site-packages (3.2.1)\r\n",
      "Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /Users/cfregly/miniconda3/envs/localsm/lib/python3.7/site-packages (from matplotlib) (2.4.7)\r\n",
      "Requirement already satisfied: python-dateutil>=2.1 in /Users/cfregly/miniconda3/envs/localsm/lib/python3.7/site-packages (from matplotlib) (2.8.1)\r\n",
      "Requirement already satisfied: kiwisolver>=1.0.1 in /Users/cfregly/miniconda3/envs/localsm/lib/python3.7/site-packages (from matplotlib) (1.2.0)\r\n",
      "Requirement already satisfied: numpy>=1.11 in /Users/cfregly/miniconda3/envs/localsm/lib/python3.7/site-packages (from matplotlib) (1.18.1)\r\n",
      "Requirement already satisfied: cycler>=0.10 in /Users/cfregly/miniconda3/envs/localsm/lib/python3.7/site-packages (from matplotlib) (0.10.0)\r\n",
      "Requirement already satisfied: six>=1.5 in /Users/cfregly/miniconda3/envs/localsm/lib/python3.7/site-packages (from python-dateutil>=2.1->matplotlib) (1.14.0)\r\n"
     ]
    }
   ],
   "source": [
    "!pip install matplotlib"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "25000 train sequences\n",
      "25000 test sequences\n",
      "x_train shape: (25000, 400)\n",
      "x_test shape: (25000, 400)\n"
     ]
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "\n",
    "import logging\n",
    "logging.getLogger(\"tensorflow\").setLevel(logging.ERROR)\n",
    "import numpy as np\n",
    "import os\n",
    "\n",
    "from tensorflow.keras.preprocessing import sequence\n",
    "from tensorflow.python.keras.datasets import imdb\n",
    "\n",
    "max_features = 20000\n",
    "maxlen = 400\n",
    "\n",
    "(x_train, y_train), (x_test, y_test) = imdb.load_data(num_words=max_features)\n",
    "print(len(x_train), 'train sequences')\n",
    "print(len(x_test), 'test sequences')\n",
    "\n",
    "x_train = sequence.pad_sequences(x_train, maxlen=maxlen)\n",
    "x_test = sequence.pad_sequences(x_test, maxlen=maxlen)\n",
    "print('x_train shape:', x_train.shape)\n",
    "print('x_test shape:', x_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training data for both Local Mode training and hosted Training must be saved as files.  Accordingly, we'll save the padded data to files, locally for now, and later to Amazon S3."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_dir = os.path.join(os.getcwd(), 'data')\n",
    "os.makedirs(data_dir, exist_ok=True)\n",
    "\n",
    "train_dir = os.path.join(os.getcwd(), 'data/train')\n",
    "os.makedirs(train_dir, exist_ok=True)\n",
    "\n",
    "test_dir = os.path.join(os.getcwd(), 'data/test')\n",
    "os.makedirs(test_dir, exist_ok=True)\n",
    "\n",
    "csv_test_dir = os.path.join(os.getcwd(), 'data/csv-test')\n",
    "os.makedirs(csv_test_dir, exist_ok=True)\n",
    "\n",
    "np.save(os.path.join(train_dir, 'x_train.npy'), x_train)\n",
    "np.save(os.path.join(train_dir, 'y_train.npy'), y_train)\n",
    "np.save(os.path.join(test_dir, 'x_test.npy'), x_test)\n",
    "np.save(os.path.join(test_dir, 'y_test.npy'), y_test)\n",
    "np.savetxt(os.path.join(csv_test_dir, 'csv-test.csv'), np.array(x_test[:100], dtype=np.int32), fmt='%d', delimiter=\",\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Local Mode Training & Git Integration\n",
    "\n",
    "Amazon SageMaker’s Local Mode training feature is a convenient way to make sure your code is working as expected before moving on to full scale, hosted training. With Local Mode, you can run quick tests with just a sample of training data, and/or a small number of epochs (passes over the full training set), while avoiding the time and expense of attempting full scale hosted training using possibly buggy code.  \n",
    "\n",
    "### Prepare for Local Mode\n",
    "\n",
    "To train in Local Mode, it is necessary to have docker-compose or nvidia-docker-compose (for GPU) installed. Running the following script will install docker-compose or nvidia-docker-compose and configure the notebook environment for you."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/bin/sh: wget: command not found\n",
      "/bin/sh: wget: command not found\n",
      "/bin/bash: ./local_mode_setup.sh: No such file or directory\n"
     ]
    }
   ],
   "source": [
    "!wget -q https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-script-mode/master/local_mode_setup.sh\n",
    "!wget -q https://raw.githubusercontent.com/aws-samples/amazon-sagemaker-script-mode/master/daemon.json    \n",
    "!/bin/bash ./local_mode_setup.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Git Configuration\n",
    "\n",
    "Now we need a training script that can be used to train the model in Amazon SageMaker.  In this example, we'll use Git integration. That is, you can specify a training script that is stored in a GitHub, AWS CodeCommit or another Git repository as the entry point so that you don't have to download the scripts locally. If you do so, any source directory and dependencies should be in the same repo if they are needed.\n",
    "\n",
    "To use Git integration, pass a dict `git_config` as a parameter when you create an Amazon SageMaker Estimator object. In the `git_config` parameter, you specify the fields `repo`, `branch` and `commit` to locate the specific repo you want to use. If you do not specify `commit` in `git_config`, the latest commit of the specified repo and branch will be used by default.  Also, if authentication is required to access the repo, you can specify fields `2FA_enabled`, `username`, `password` and `token` accordingly.\n",
    "\n",
    "The script that we will use in this example is stored a public GitHub repo so we don't need authentication to access it. Let's specify the `git_config` argument here:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "git_config = {'repo': 'https://github.com/aws-samples/amazon-sagemaker-script-mode', \n",
    "              'branch': 'master'}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Local Mode Estimator setup\n",
    "\n",
    "The next step is to set up a TensorFlow Estimator for Local Mode training. The Estimator will be used to configure and start a training job.  A key parameters for the Estimator is the `train_instance_type`, which is the kind of hardware on which training will run. In the case of Local Mode, we simply set this parameter to `local_gpu` to invoke Local Mode training on the GPU, or to `local` if the instance has a CPU. Other parameters of note are the algorithm’s hyperparameters, which are passed in as a dictionary, and a Boolean parameter indicating that we are using Script Mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "from sagemaker.tensorflow import TensorFlow\n",
    "\n",
    "\n",
    "model_dir = '/opt/ml/model'\n",
    "train_instance_type = 'local'\n",
    "hyperparameters = {'epochs': 1, 'batch_size': 64, 'learning_rate': 0.01}\n",
    "local_estimator = TensorFlow(\n",
    "                       git_config=git_config,\n",
    "                       source_dir='tf-sentiment-script-mode',\n",
    "                       entry_point='sentiment.py',\n",
    "                       model_dir=model_dir,\n",
    "                       train_instance_type=train_instance_type,\n",
    "                       train_instance_count=1,\n",
    "                       hyperparameters=hyperparameters,\n",
    "                       role='arn:aws:iam::835319576252:role/service-role/AmazonSageMaker-ExecutionRole-20191006T135881',\n",
    "                       base_job_name='tf-sentiment',\n",
    "                       framework_version='2.1',\n",
    "                       py_version='py3',\n",
    "                       script_mode=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we'll briefly train the model in Local Mode.  Since this is just to make sure the code is working, we'll train for only one epoch.  On a notebook instance using a general purpose CPU-based instance type such as m5.xlarge, this one epoch will take about 3 or 4 minutes.  As you'll see from the logs below the cell when training is complete, even when trained for only one epoch, the accuracy of the model on training data is around 80%.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "ename": "ClientError",
     "evalue": "An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mClientError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-23-dde084cfed9b>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m           'test': f'file://{test_dir}'}\n\u001b[1;32m      3\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m \u001b[0mlocal_estimator\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/tensorflow/estimator.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, inputs, wait, logs, job_name, experiment_config, run_tensorboard_locally)\u001b[0m\n\u001b[1;32m    483\u001b[0m                 \u001b[0mtensorboard\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    484\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 485\u001b[0;31m             \u001b[0mfit_super\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    486\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    487\u001b[0m     \u001b[0;34m@\u001b[0m\u001b[0mclassmethod\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/tensorflow/estimator.py\u001b[0m in \u001b[0;36mfit_super\u001b[0;34m()\u001b[0m\n\u001b[1;32m    462\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    463\u001b[0m         \u001b[0;32mdef\u001b[0m \u001b[0mfit_super\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 464\u001b[0;31m             \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mTensorFlow\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mwait\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mlogs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mjob_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexperiment_config\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    465\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    466\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mrun_tensorboard_locally\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mwait\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/estimator.py\u001b[0m in \u001b[0;36mfit\u001b[0;34m(self, inputs, wait, logs, job_name, experiment_config)\u001b[0m\n\u001b[1;32m    490\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    491\u001b[0m         \"\"\"\n\u001b[0;32m--> 492\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_prepare_for_training\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjob_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mjob_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    493\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    494\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mlatest_training_job\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_TrainingJob\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstart_new\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0minputs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mexperiment_config\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/estimator.py\u001b[0m in \u001b[0;36m_prepare_for_training\u001b[0;34m(self, job_name)\u001b[0m\n\u001b[1;32m   1661\u001b[0m                 \u001b[0mconstructor\u001b[0m \u001b[0;32mif\u001b[0m \u001b[0mapplicable\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1662\u001b[0m         \"\"\"\n\u001b[0;32m-> 1663\u001b[0;31m         \u001b[0msuper\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mFramework\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_prepare_for_training\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mjob_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mjob_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1664\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1665\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgit_config\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/estimator.py\u001b[0m in \u001b[0;36m_prepare_for_training\u001b[0;34m(self, job_name)\u001b[0m\n\u001b[1;32m    357\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutput_path\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    358\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 359\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0moutput_path\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"s3://{}/\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msagemaker_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdefault_bucket\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    360\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    361\u001b[0m         \u001b[0;31m# Prepare rules and debugger configs for training.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/session.py\u001b[0m in \u001b[0;36mdefault_bucket\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    368\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mdefault_bucket\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    369\u001b[0m             account = self.boto_session.client(\n\u001b[0;32m--> 370\u001b[0;31m                 \u001b[0;34m\"sts\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mregion_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mregion\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mendpoint_url\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msts_regional_endpoint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mregion\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    371\u001b[0m             ).get_caller_identity()[\"Account\"]\n\u001b[1;32m    372\u001b[0m             \u001b[0mdefault_bucket\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"sagemaker-{}-{}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mregion\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maccount\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_api_call\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    314\u001b[0m                     \"%s() only accepts keyword arguments.\" % py_operation_name)\n\u001b[1;32m    315\u001b[0m             \u001b[0;31m# The \"self\" in this scope is referring to the BaseClient.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 316\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_api_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moperation_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    317\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    318\u001b[0m         \u001b[0m_api_call\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpy_operation_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_make_api_call\u001b[0;34m(self, operation_name, api_params)\u001b[0m\n\u001b[1;32m    633\u001b[0m             \u001b[0merror_code\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparsed_response\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Error\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Code\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    634\u001b[0m             \u001b[0merror_class\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfrom_code\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merror_code\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 635\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0merror_class\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparsed_response\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moperation_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    636\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    637\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mparsed_response\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mClientError\u001b[0m: An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid."
     ]
    }
   ],
   "source": [
    "inputs = {'train': f'file://{train_dir}',\n",
    "          'test': f'file://{test_dir}'}\n",
    "\n",
    "local_estimator.fit(inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#  Hosted Training\n",
    "\n",
    "After we've confirmed our code is working using Local Mode training, we can move on to use SageMaker's hosted training, which typically uses more powerful compute resources separate from your less powerful, lower cost notebook instance.  Hosted training spins up one or more instances (i.e. a cluster) for training, and then tears the cluster down when training is complete. In general, hosted training is preferred for doing actual large-scale training on more powerful instances, especially for distributed training on a single instance with multiple GPUs or multiple instances each having multiple GPUs. \n",
    "\n",
    "### Upload data to S3\n",
    "\n",
    "Before starting hosted training, the data must be present in storage that can be accessed by SageMaker. The storage options are:  Amazon S3 (object storage service), Amazon EFS (elastic NFS file system service), and Amazon FSx for Lustre (high-performance file system service). For this example, we'll upload the data to S3.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'upload_data' method will be deprecated in favor of 'S3Uploader' class (https://sagemaker.readthedocs.io/en/stable/s3.html#sagemaker.s3.S3Uploader) in SageMaker Python SDK v2.\n"
     ]
    },
    {
     "ename": "ClientError",
     "evalue": "An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mClientError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-24-301b7a0977f8>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      6\u001b[0m \u001b[0mtestdata_s3_prefix\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'{}/data/test'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0ms3_prefix\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m \u001b[0mtrain_s3\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msagemaker\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSession\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupload_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'./data/train/'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey_prefix\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtraindata_s3_prefix\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m \u001b[0mtest_s3\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msagemaker\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSession\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupload_data\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpath\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'./data/test/'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkey_prefix\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mtestdata_s3_prefix\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/session.py\u001b[0m in \u001b[0;36mupload_data\u001b[0;34m(self, path, bucket, key_prefix, extra_args)\u001b[0m\n\u001b[1;32m    205\u001b[0m             \u001b[0mkey_suffix\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mname\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    206\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 207\u001b[0;31m         \u001b[0mbucket\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbucket\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdefault_bucket\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    208\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0ms3_resource\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    209\u001b[0m             \u001b[0ms3\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mboto_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mresource\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"s3\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mregion_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mboto_region_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/session.py\u001b[0m in \u001b[0;36mdefault_bucket\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    368\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mdefault_bucket\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    369\u001b[0m             account = self.boto_session.client(\n\u001b[0;32m--> 370\u001b[0;31m                 \u001b[0;34m\"sts\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mregion_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mregion\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mendpoint_url\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msts_regional_endpoint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mregion\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    371\u001b[0m             ).get_caller_identity()[\"Account\"]\n\u001b[1;32m    372\u001b[0m             \u001b[0mdefault_bucket\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m\"sagemaker-{}-{}\"\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mregion\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0maccount\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_api_call\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    314\u001b[0m                     \"%s() only accepts keyword arguments.\" % py_operation_name)\n\u001b[1;32m    315\u001b[0m             \u001b[0;31m# The \"self\" in this scope is referring to the BaseClient.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 316\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_api_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moperation_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    317\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    318\u001b[0m         \u001b[0m_api_call\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpy_operation_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_make_api_call\u001b[0;34m(self, operation_name, api_params)\u001b[0m\n\u001b[1;32m    633\u001b[0m             \u001b[0merror_code\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparsed_response\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Error\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Code\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    634\u001b[0m             \u001b[0merror_class\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfrom_code\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merror_code\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 635\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0merror_class\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparsed_response\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moperation_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    636\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    637\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mparsed_response\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mClientError\u001b[0m: An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid."
     ]
    }
   ],
   "source": [
    "import sagemaker\n",
    "\n",
    "s3_prefix = 'tf-keras-sentiment'\n",
    "\n",
    "traindata_s3_prefix = '{}/data/train'.format(s3_prefix)\n",
    "testdata_s3_prefix = '{}/data/test'.format(s3_prefix)\n",
    "\n",
    "train_s3 = sagemaker.Session().upload_data(path='./data/train/', key_prefix=traindata_s3_prefix)\n",
    "test_s3 = sagemaker.Session().upload_data(path='./data/test/', key_prefix=testdata_s3_prefix)\n",
    "\n",
    "inputs = {'train':train_s3, 'test': test_s3}\n",
    "print(inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Hosted training Estimator setup\n",
    "\n",
    "With the training data now in S3, we're ready to set up an Estimator object for hosted training. It is similar to the Local Mode Estimator, except the `train_instance_type` has been set to a ML instance type instead of a local type for Local Mode. Additionally, we've increased the number of epochs for actual training, as opposed to just testing the code in Local Mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "ename": "ClientError",
     "evalue": "An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mClientError\u001b[0m                               Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-7-81aa79ac4438>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m     13\u001b[0m                        \u001b[0mtrain_instance_count\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     14\u001b[0m                        \u001b[0mhyperparameters\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mhyperparameters\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 15\u001b[0;31m                        \u001b[0mrole\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msagemaker\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_execution_role\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     16\u001b[0m                        \u001b[0mbase_job_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'tf-sentiment'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     17\u001b[0m                        \u001b[0mframework_version\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'2.1'\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/session.py\u001b[0m in \u001b[0;36mget_execution_role\u001b[0;34m(sagemaker_session)\u001b[0m\n\u001b[1;32m   3339\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0msagemaker_session\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3340\u001b[0m         \u001b[0msagemaker_session\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mSession\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 3341\u001b[0;31m     \u001b[0marn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msagemaker_session\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget_caller_identity_arn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   3342\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   3343\u001b[0m     \u001b[0;32mif\u001b[0m \u001b[0;34m\":role/\"\u001b[0m \u001b[0;32min\u001b[0m \u001b[0marn\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/sagemaker/session.py\u001b[0m in \u001b[0;36mget_caller_identity_arn\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m   2936\u001b[0m             \u001b[0;34m\"sts\"\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2937\u001b[0m             \u001b[0mregion_name\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mboto_region_name\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2938\u001b[0;31m             \u001b[0mendpoint_url\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0msts_regional_endpoint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mboto_region_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m,\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2939\u001b[0m         ).get_caller_identity()[\"Arn\"]\n\u001b[1;32m   2940\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_api_call\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    314\u001b[0m                     \"%s() only accepts keyword arguments.\" % py_operation_name)\n\u001b[1;32m    315\u001b[0m             \u001b[0;31m# The \"self\" in this scope is referring to the BaseClient.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 316\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_make_api_call\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0moperation_name\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    317\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    318\u001b[0m         \u001b[0m_api_call\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m__name__\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mstr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpy_operation_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m~/miniconda3/envs/localsm/lib/python3.7/site-packages/botocore/client.py\u001b[0m in \u001b[0;36m_make_api_call\u001b[0;34m(self, operation_name, api_params)\u001b[0m\n\u001b[1;32m    633\u001b[0m             \u001b[0merror_code\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mparsed_response\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Error\"\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m{\u001b[0m\u001b[0;34m}\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Code\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    634\u001b[0m             \u001b[0merror_class\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfrom_code\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0merror_code\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 635\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0merror_class\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mparsed_response\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moperation_name\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    636\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    637\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mparsed_response\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mClientError\u001b[0m: An error occurred (InvalidClientTokenId) when calling the GetCallerIdentity operation: The security token included in the request is invalid."
     ]
    }
   ],
   "source": [
    "from sagemaker.tensorflow import TensorFlow\n",
    "\n",
    "model_dir = '/opt/ml/model'\n",
    "train_instance_type = 'ml.p3.8xlarge'\n",
    "hyperparameters = {'epochs': 10, 'batch_size': 256, 'learning_rate': 0.01}\n",
    "\n",
    "estimator = TensorFlow(\n",
    "                       git_config=git_config,\n",
    "                       source_dir='tf-sentiment-script-mode',\n",
    "                       entry_point='sentiment.py',\n",
    "                       model_dir=model_dir,\n",
    "                       train_instance_type=train_instance_type,\n",
    "                       train_instance_count=1,\n",
    "                       hyperparameters=hyperparameters,\n",
    "                       role=sagemaker.get_execution_role(),\n",
    "                       base_job_name='tf-sentiment',\n",
    "                       framework_version='2.1',\n",
    "                       py_version='py3',\n",
    "                       script_mode=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Distributed training on a single multi-GPU instance\n",
    "\n",
    "The instance type selected above, p3.8xlarge, contains four GPUs based on NVIDIA's V100 Tensor Core architecture.  This presents an opportunity to do distributed training within a single multi-GPU instance, utilizing all four GPUs to reduce total training time compared to using a single GPU.  Although using multiple instances also is a possibility, using a single multi-GPU instance may be more performant because it avoids extra network traffic necessary to coordinate multiple instances.  For larger datasets and more complex models, using multiple instances may be a necessity, however, that is not the case here.\n",
    "\n",
    "To utilize all four GPUs on the instance, you don't need to do anything special in Amazon SageMaker:  TensorFlow 2 itself will handle the details under the hood.  TensorFlow 2 includes several native distribution strategies, including MirroredStrategy, which is well-suited for training a model using multiple GPUs on a single instance.  To enable MirroredStrategy, simply add the following lines of code in your training script before you define and compile the model:\n",
    "\n",
    "```python\n",
    "def get_model(learning_rate):\n",
    "\n",
    "    mirrored_strategy = tf.distribute.MirroredStrategy()\n",
    "    \n",
    "    with mirrored_strategy.scope():\n",
    "        embedding_layer = tf.keras.layers.Embedding(max_features,\n",
    "                                                    embedding_dims,\n",
    "                                                    input_length=maxlen)\n",
    "        ....\n",
    "        model.compile(loss='binary_crossentropy',\n",
    "                      optimizer=optimizer,\n",
    "                      metrics=['accuracy'])\n",
    "    \n",
    "    return model\n",
    "```\n",
    "\n",
    "Additionally, the batch size was increased in the Estimator hyperparameters to account for the fact that batches are divided among multiple GPUs.  If you are interested in reviewing the rest of the training code, it is at the GitHub repository referenced above in the `git_config` variable.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Start the hosted training job\n",
    "\n",
    "With the change in training instance type and increase in epochs, we simply call `fit` to start the actual hosted training.  The training job should take around 5 minutes, including the time needed to spin up the training instance.  At the end of hosted training, you'll see from the logs below the cell that accuracy on the training set has greatly increased, and accuracy on the validation set is approaching 90%.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "estimator.fit(inputs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Although the training accuracy has improved, the validation accuracy plateaued, so the model might be overfitting now:  it could be less able to generalize to data it has not yet seen.  This is the case even though we are employing dropout as a regularization technique to reduce the possibility of overfitting.  (See the training script at the GitHub repository referenced above.)  For a production model, further experimentation would be necessary.\n",
    "\n",
    "TensorFlow 2's tf.keras API provides a convenient way to capture the history of model training.  When the model was saved after training, the history was saved alongside it.  To retrieve the history, we first download the trained model from the S3 bucket where SageMaker stored it.  Models trained by SageMaker are always accessible in this way to be run anywhere.  Next, we can unzip it to gain access to the history data structure, and then simply load the history as JSON:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json \n",
    "\n",
    "!aws s3 cp {estimator.model_data} ./model/model.tar.gz\n",
    "!tar -xzf ./model/model.tar.gz -C ./model\n",
    "\n",
    "with open('./model/history.p', \"r\") as f:\n",
    "    history = json.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we can plot the training curves based on the history, with separate graphs for model accuracy and model loss.  We can see that training converged relatively smoothly to higher model accuracy and correspondingly lower model loss."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def plot_training_curves(history): \n",
    "    fig, axes = plt.subplots(1, 2, figsize=(12, 4), sharex=True)\n",
    "    ax = axes[0]\n",
    "    ax.plot(history['accuracy'], label='train')\n",
    "    ax.set(title='model accuracy', ylabel='accuracy', xlabel='epoch')\n",
    "    ax.legend()\n",
    "    ax = axes[1]\n",
    "    ax.plot(history['loss'], label='train')\n",
    "    ax.set(title='model loss', ylabel='loss', xlabel='epoch')\n",
    "    ax.legend()\n",
    "    fig.tight_layout()\n",
    "    \n",
    "plot_training_curves(history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Batch Prediction\n",
    "\n",
    "\n",
    "If our use case requires individual predictions in near real-time, SageMaker hosted endpoints can be created. Hosted endpoints also can be used for pseudo-batch prediction, but the process is more involved than simply using SageMaker's Batch Transform feature, which is designed for large-scale, asynchronous batch inference.\n",
    "\n",
    "To use Batch Transform, we first upload to S3 some input test data to be transformed.  The data can be in any format accepted by your model; in this case, it is CSV."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "csvtestdata_s3_prefix = '{}/data/csv-test'.format(s3_prefix)\n",
    "csvtest_s3 = sagemaker.Session().upload_data(path='./data/csv-test/', key_prefix=csvtestdata_s3_prefix)\n",
    "print(csvtest_s3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Transformer object must be set up to describe the Batch Transform job, including the amount and type of inference hardware to be used.  Then the actual transform job itself is started with a call to the `transform` method of the Transformer.  When setting up Batch Transform jobs, hardware considerations are important, just as they are for training:\n",
    "\n",
    "- `instance_count`:  Batch Transform can spin up a cluster of multiple instances; at the end of the job, the cluster is torn down automatically.  Since this dataset is small, we'll just use one instance.\n",
    "- `instance_type`:  When doing inference for smaller models, such as this one, often CPU-based instance types can be used instead of GPU instance types, allowing significant cost savings.  Note, however, that the choice of specific CPU instance type can significantly affect inference speed:  although we could use a general purpose instance here such as a m5.xlarge, if we use a compute-optimized c5.xlarge instance, the total batch inference time is cut in half."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "transformer = estimator.transformer(instance_count=1, \n",
    "                                    instance_type='ml.c5.xlarge')\n",
    "\n",
    "transformer.transform(csvtest_s3, content_type='text/csv')\n",
    "print('Waiting for transform job: ' + transformer.latest_transform_job.job_name)\n",
    "transformer.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now download the batch predictions from S3 to the local filesystem on the notebook instance; the predictions are contained in a file with a .out extension, and are embedded in JSON.  Next we'll load the JSON and examine the predictions, which are confidence scores from 0.0 to 1.0 where numbers close to 1.0 indicate positive sentiment, while numbers close to 0.0 indicate negative sentiment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "batch_output = transformer.output_path\n",
    "!mkdir -p batch_data/output\n",
    "!aws s3 cp --recursive $batch_output/ batch_data/output/\n",
    "\n",
    "with open('batch_data/output/csv-test.csv.out', 'r') as f:\n",
    "    jstr = json.load(f)\n",
    "    results = [float('%.3f'%(item)) for sublist in jstr['predictions'] for item in sublist]\n",
    "    print(results)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's look at the text of some actual reviews to see the predictions in action.  First, we have to convert the integers representing the words back to the words themselves by using a reversed dictionary.  Next we can decode the reviews, taking into account that the first 3 indices were reserved for \"padding\", \"start of sequence\", and \"unknown\", and removing a string of unknown tokens from the start of the review."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import re\n",
    "\n",
    "regex = re.compile(r'^[\\?\\s]+')\n",
    "\n",
    "word_index = imdb.get_word_index()\n",
    "reverse_word_index = dict([(value, key) for (key, value) in word_index.items()])\n",
    "first_decoded_review = ' '.join([reverse_word_index.get(i - 3, '?') for i in x_test[0]])\n",
    "regex.sub('', first_decoded_review)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Overall, this review looks fairly negative.  Let's compare the actual label with the prediction:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_sentiment(score):\n",
    "    return 'positive' if score > 0.5 else 'negative' \n",
    "\n",
    "print('Labeled sentiment for this review is {}, predicted sentiment is {}'.format(get_sentiment(y_test[0]), \n",
    "                                                                                  get_sentiment(results[0])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Training deep learning models is a stochastic process, so your results may vary -- there is no guarantee that the predicted result will match the actual label. However, it is likely that the sentiment prediction agrees with the label for this review.  Let's now examine another review:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "second_decoded_review = ' '.join([reverse_word_index.get(i - 3, '?') for i in x_test[5]])\n",
    "regex.sub('', second_decoded_review)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Labeled sentiment for this review is {}, predicted sentiment is {}'.format(get_sentiment(y_test[5]), \n",
    "                                                                                  get_sentiment(results[5])))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, it is likely (but not guaranteed) that the prediction agreed with the label for the test data.  Note that there is no need to clean up any Batch Transform resources:  after the transform job is complete, the cluster used to make inferences is torn down.  Now that we've reviewed some sample predictions as a sanity check, this brief example is complete.   "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (localsm)",
   "language": "python",
   "name": "localsm"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
